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This  study  project  report  represents  the  views  and  conclusions  and  recom- 
mendations of  the  author  and  does  not  necessarily  reflect  the  official 
opinion  of  the  Defense  Systems  Management  School  or  the  Department  of 
Defense. 
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EXECUTIVE  SUMMARY 


The  overall  purpose  of  this  project  was  to  investigate  the  reasons 
for  interface  problems  between  the  U.  S.  Army  Operational  Test  and 
Evaluation  Agency  (OTEA)  and  the  Army  Material  Development  Community 
(AMDC),  as  represented  by  U.  S.  Army  Material  Development  & Readiness 
Command  (DARCOM)  at  the  headquarters  and  project  manager  levels,  and 
to  ascertain  if  suggestions  could  be  made  to  reduce  or  eliminate  these 
difficulties.  The  author  had  noted  the  existence  of  significant  OTEA-AMDC 
interface  problems  during  a three  year  tenure  with  the  former  organization. 


The  methodology  employed  consisted  of  a literature  search  into  the 
history  of  Army  operational  testing  and  evaluation,  and  both  structured 
and  informal  interviews  with  persons  either  currently  or  recently  as- 
signed to  OTEA,  Headquarters,  DARCOM,  or  project  management  offices. 

The  study  revealed  that  negative  perceptions  related  to  historical 
and  situational  (environmental)  factors  are  the  key  contributors  to 
the  interface  problems  and  that  the  situation  is  more  acute  at  the  working 
levels  than  at  the  managerial  layers  of  the  three  organizations  (OTEA, 
Headquarters,  DARCOM  and  the  PMO's). 

The  author  recommends  the  use  of  balanced  in-house  training  pro- 
grams, better  distribution  of  the  personnel  resources  in  the  Project 
Management  Development  Program,  and  increased  opportunity  for  assigned 
personnel  to  attend  the  Program  Management  Course  at  the  Defense  Systems 
Management  School  (DSMS)  — as  approaches  which  may  reduce  or  alleviate 
the  interface  problems. 
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SECTION  I 
INTRODUCTION 

Purpose  of  the  Study  Project 

In  February  1973,  the  author,  with  a newly  acquired  masters  de- 
gree in  hand,  was  assigned  to  a utilization  tour  with  a fledging  new 
operating  field  element  of  the  Army  staff— 0TEA--then  located  at  Fort 
Belvoir,  Virginia.  The  subsequent  three  years  with  that  organization 
were  most  rewarding  in  terms  of  professional  challenge  and  personal 
satisfaction.  A feeling  of  accomplishment  was  felt  by  all  personnel  as 
the  agency  progressed  from  a young,  struggling  stepchild  to  a fully 
accepted  and  powerful  member  of  the  Army  staff  hierarchy. 

Soon  after  being  assigned  to  OTEA,  it  became  evident  to  the  writer 
that  the  agency  generally  interfaced  with  three  organizational  entities 
in  the  conduct  of  its  business.  They  were  the  Army  staff,  the  U.  S. 

Army  Training  and  Doctrine  Command,  and  the  Army  Material  Command  (AMC). 
In  the  case  of  the  latter,  most  of  the  contacts  were  at  the  headquarters 
and  project  management  office  levels.  Additionally,  it  was  quite  notice- 
able that  many  of  the  relations  with  the  AMC  organization  were  con- 
ducted in  a strained  atmosphere.  These  relations  continued,  though  to 
a lesser  degree,  throughout  the  author's  three  years  at  OTEA. 
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Thus,  when  the  curriculum  in  the  Program  Management  Course  (PMC) 
at  the  DSMS  afforded  an  opportunity  to  conduct  research  on  a pertinent 
topic  in  the  field  of  material  acquisition,  the  author  selected  the 
paper's  topic  with  the  hope  that  a very  real  problem  (at  least  to  the 
Army)  could  be  analyzed. 

In  light  of  the  above  reasons,  the  purpose  of  this  study  is  to  in- 
vestigate the  reasons  for  interface  problems  between  OTEA  and  the  Army 
Material  Development  Community  (identified  henceforth  as  AMDC)  as  rep- 
resented by  the  Army  Development  and  Readiness  Command  (DARCOM),  which 
succeeded  AMC,  at  the  headquarters  and  project  manager  level,  and  to 
ascertain  if  suggestions  can  be  made  to  alleviate  these  difficulties. 

Goals 

The  goals  of  this  study  are: 

1.  To  identify  the  reasons  for  interface  problems  between  OTEA 
and  DARCOM  as  perceived  at  OTEA  and  at  the  headquarters  and  project 
manager  levels  of  the  latter; 

2.  To  evaluate  the  resultant  impact  on  both  organizations; 

3.  To  make  recommendations  regarding  resolution  of  the  prob- 
lems, when  possible. 

Scope  and  Limitations 

This  study  effort  included  a literature  research  into  the  back- 
ground and  history  of  operational  test  and  evaluation  in  the  Army  and 
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the  conducting  of  a series  of  formal  interviews  with  personnel  assigned 
to  OTEA,  Headquarters,  DARCOM  and  various  project  management  offices  (see 
Appendices  A and  B).  Additionally,  the  interviews  were  supplemented  with 
numerous  discussions  with  members  of  Program  Management  Course  76-1.  All 
persons  who  were  interviewed  were  assured  that  their  comments  and  opinions 
would  be  treated  in  a confidential  manner. 


SECTION  II 


OVERVIEW  OF  ARMY  OPERATIONAL  TESTING 

Terminology 

The  following  basic  terms  are  used  in  this  report  and  are  defined 
as  follows: 

User  Testing:  A generic  term  encompassing  Operational  Testing 

(OT)  and  Force  Development  Testing  and  Experimentation  (FDTE).  (3)* 

Joint  User  Testing:  Testing  in  which  the  Army  participates 

with  one  or  more  of  the  Services  to  evaluate  systems  or  concepts  having 
an  interface  with  or  requiring  a test  environment  of  another  service.  (3) 
Operational  Testing  (OT):  OT  is  that  test  and  evaluation  con- 

ducted to  estimate  the  prospective  system's  military  utility,  operational 
effectiveness,  and  operational  suitability  (including  compatibility; 
interoperability;  reliability,  availability,  and  maintainability  (RAM); 
and  logistic  supportability;  operational  man  (soldier)-machine  interface 
and  training  requirements),  and  need  for  any  modifications.  In  addition, 
OT  provides  information  on  organization,  personnel  requirements,  doctrine, 
and  tactics.  Also  it  may  provide  data  to  support  or  verify  operating  in- 
structions, publications,  and  handbooks.  OT  will  be  accomplished  by 
operational  and  support  personnel  of  the  type  and  qualifications  of 
those  expected  to  use  and  maintain  the  system  when  deployed,  and  will 
be  conducted  in  as  realistic  an  operational  environment  as  possible.  (2) 


*This  notation  will  be  used  throughout  the  report  in  referencing 
documents  listed  in  the  Bibliography. 
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Development  Testing  (DT):  DT  is  that  test  and  evaluation 

conducted  to  demonstrate  that  the  engineering  design  and  development 
process  is  complete;  to  demonstrate  that  the  design  risks  have  been  mini- 
mized; to  demonstrate  that  the  system  will  meet  specifications;  and  to 
estimate  the  system's  military  utility  when  introduced.  DT  is  planned, 
conducted,  and  monitored  by  the  materiel  developer  and  the  results  thereof 
are  reported  by  that  agency  directly  to  the  Army  System  Acquisition  Re- 
view Council  (ASARC)  or  In-Process  Review  ( IPR) . (2) 

Force  Development  Testing  and  Experimentation:  Tests,  ranging 

from  the  small,  highly  instrumented , high  resolution  field  experiment, 
to  the  large,  less  instrumented,  controlled  scenario,  low  resolution 
field  test,  (a)  yield  data  which  are  evaluated  largely  by  using  sub- 
jective rather  than  analytical  techniques,  and  (b)  are  conducted  to  eval- 
uate new  concepts  of  tactics,  doctrine,  organization  and  new  items  of 
materiel.  (3) 

Background. 

The  concept  of  sequential  and  separately  conducted  engineer  tests 
and  service  (user)  tests  of  Army  materiel  systems  was  originally  es- 
tablished in  1924.  Even  though  the  identity,  titles,  and  organizational 
structures  of  developers  and  users  underwent  many  changes  between  1924 
and  1962,  the  basic  concept  of  independent  developer  and  user  testing 
did  not  change.  Increased  test  and  evaluation  lead  time  in  the  late 
1950's  caused  considerable  concern,  and  was  attributed  primarily  to  two 
factors:  (1)  developer  (technical  services)  and  user  test  duplication, 
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redundancy,  and  conflict;  and  (2)  lack  of  valid  operational  (user) 
test  methodology.  These  lead  time  problems  and  other  considerations 
were  closely  associated  with  a major  Army  reorganization  in  1962  which 
resulted  in  the  consolidation  of  all  Army  test  and  evaluation  under  the 
developer,  Army  Materiel  Command  (AMC).  In  turn,  the  U.  S.  Army  Test 
and  Evaluation  Command  (TECOM)  was  established  in  1962  under  AMC  to  con- 
duct all  engineering  and  service  (user)  tests  for  Army  materiel.  (14) 

During  its  first  four  years  of  operation  TECOM  was  successful  in 
consolidating  facilities  and  reducing  lead  time  in  testing.  However, 
there  was  a general  recognition  that  material/engineering  testing  was 
being  emphasized  at  the  expense  of  operational  considerations  and  testing. 
This  situation  led  to  a Department  of  the  Army  inquiry  into  its  testing 
and  evaluation  program.  This  effort,  undertaken  in  1966,  was  labeled 
A Study  of  Army  Test  and  Evaluation  (SATE). 

SATE  revealed  two  important  deficiencies  of  Army  test  and  evaluation: 
(1)  as  a result  of  the  1962  reorganization  there  was  no  independent  check 
or  evaluation  by  the  user,  the  U.  S.  Army  Combat  Developments  Command  (CDC); 
and  (2)  there  was  a lack  of  sufficient  operational  (user)  test  methodology 
available  to  enable  the  proper  conduct  of  effective  and  meaningful  ser- 
vice (user)  tests.  Because  of  SATE,  in  1966,  the  Army  Chief  of  Staff  di- 
rected that  TECOM  continue  to  develop  improved  test  methodology  for  the  ser- 
vice (user)  test;  that  CDC  continue  to  be  responsible  for  overall  evaluation 
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of  the  military  worth  of  Army  materiel,  and  that  CDC  approve  all  TECOM 
service  (user)  test  plans.  Although  SATE  (1966)  recognized  the  absence 
of  user  influence  on  Army  test  and  evaluation,  it  did  not  recommend  re- 
organization as  a remedy.  (14) 

Army  test  and  evaluation  continued  to  receive  a great  deal  of 
criticism  in  the  late  '/960's,  as  did  other  facets  of  the  Department  of 
Defense  (DOD)  materiel  acquisition  system.  Events  began  to  culminate, 
however,  when  the  1970  Blue  Ribbon  Defense  Panel  found  that  the  DOD 
testing  apparatus  as  a whole  did  not  have  an  independent  check  or  evalu- 
ation capability  and  that  the  military  services  were  not  conducting 
sufficient  operational  testing.  The  general  consensus  was  that  faulty 
requirements  which  conflicted  with  organization  and  doctrine  would  not 
be  detected  in  the  absence  of  an  independent  check  or  evaluation  and 
would  consequently  result  in  the  creation  and  production  of  costly  but 
ineffective  weapon  systems.  (14) 

The  panel's  special  Operation  Test  and  Evaluation  ( OT&E ) Group 
also  concluded  that  "OT&E  within  the  Services  is  done  most  effectively 
when  OT&E  organizations  report  directly  to  the  Chief  of  the  Service, 
representing  both  the  developer  and  user,  but  organizationally  independent 
of  both.  There  are,  however,  considerable  forces  within  the  services 
which  resist  the  independence  of  the  OT&E  organization.'1  (8) 

The  overall  panel  report  to  the  President  was  specific  about  what 
it  believed  to  be  a major  flaw  in  the  Army's  approach  to  operational 
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testing  and  evaluation.  It  stated,  "The  basic  problem  with  Army  OT&E 
is  that  the  developer,  in  effect,  tests  and  evaluates  the  operational 
suitability  of  what  he  develops."  (6)  That  is,  the  Army  Materiel 
Command  developed  and  produced  materiel  through  its  commodity  commands  - 
Tank  and  Automotive  Command,  for  example  - while  testing  and  evaluating 
the  materiel  through  its  Test  and  Evaluation  Command  (TECOM). 


Subsequent  to  the  panel's  recommendations,  DOD  began  to  revise 
its  policies  and  structure  in  an  effort  to  accommodate  some  of  the 
recommendations.  In  an  11  February  1971  memorandum  to  the  Services, 
then  Deputy  Secretary  of  Defense  Packard  wrote: 

"Although  each  Service  now  has  a somewhat 
different  way  of  organizing  for  operational 
test  and  evaluation,  it  is  apparent  to  me 
that  this  function  can  best  be  performed  by 
an  agency  which  is  separate  and  distinct  from 
the  developing  command  and  which  reports  the 
results  of  its  test  and  evaluation  efforts 
directly  to  the  Chief  of  the  Service.  More- 
over, within  the  Service  headquarters  staff, 
there  needs  to  be  an  office  with  a clear 
OT&E  identification  to  provide  staff  as- 
sistance directly  to  the  Service  Chief  and 
to  provide  a headquarters  focal  point  for 

the  independent  OT&E  field  agency  

Accordingly,  each  Service  is  requested  to 
restructure  its  organization  for  OT&E  along 
the  lines  specified  above. 

As  a second  step,  I am  establishing  a Deputy 
Director  for  Test  and  Evaluation  within  ODDR&E 
with  across  the  board  responsibilities  for 
OSD  in  test  and  evaluation  matters.  This  of- 
fice will  review  and  approve  test  and  evalu- 
ation plans  prepared  by  the  Service  and  will 
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provide  an  assessment  of  results  obtained." 
(as  quoted  in  9:  Vol  II,  B-6) 


While  DOD  was,  internally,  responding  to  the  panel's  report. 
Congress,  which  had  also  scrutinized  the  panel's  deliberations  and 
advice,  increased  the  pressure  on  the  services  by  linking  appropriations 
to  operational  testing. 


Specifically,  the  Armed  Forces  Appropriation  Authorization  Act  for 

FY  1972  (PL  92-156)  included  Section  506  requiring: 

Beginning  with  1972,  the  Secretary  of  Defense 
shall  submit  to  the  Congress  each  calendar  year, 
at  the  same  time  the  President  submits  the  Bud- 
get to  the  Congress,  a written  report  regarding 
development  and  procurement  scheduled  for  each 
weapon  system  for  which  fund  authorization  is  re- 
quired and  for  which  any  funds  for  procurement 
are  requested  in  such  budget.  Beginning  with 
the  calendar  year  1973,  there  shall  be  included 
in  the  report  data  on  operational  testing  and 
evaluation  for  each  such  weapon  system  for 
which  funds  for  procurement  are  requested. 

Additionally,  the  section  required  that  the  reports  specifically 
include  "the  results  of  all  operational  testing,  or,  if  operational 
testing  and  evaluation  has  not  been  conducted,  a statement  of  the  reasons 
therefore,  and  the  results  of  such  other  testing  and  evaluation  as  has 
been  conducted."  (9:  Vol  II,  B-8) 


The  Army,  in  an  effort  to  respond  to  OSD  and  Congressional  criti- 
cism, reoriented  its  policies  and  procedures  relating  to  operational 
testing  and  evaluation  within  its  existing  organizational  structure. 
Specifically,  the  key  Army  regulation  pertaining  to  test  and  evaluation 
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(AR  70-10,  Test  and  Evaluation  During  Development  and  Acquisition  of 
Material)  was  revised  to  include  an  Expanded  Service  Test  (EST)  as  a 
replacement  for  its,  allegedly,  ineffective  predecessor,  The  Service 
Test.  The  EST,  as  conceived,  was  to  include  a field  exercise  and  more 
thorough  operational  (user)  testing  and  was  intended  to  be  the  pivotal 
point  in  Army  test  and  evaluation,  thereby  providing  the  initial  and 
only  operational  (user)  test  data  input  to  the  low  rate  production 
decision. 

The  Army's  internal  actions  satisfied  some  critics,  both  external 
and  internal.  However,  the  desire  for  an  independent  operational  test 
and  evaluation  agency  within  each  of  the  services  had  already  been 
iterated  by  Congressional  and  OSD  spokesmen  and  the  Army  acceded  in 
September  1972  by  activating  the  U.  S.  Army  Operational  Test  and  Eval- 
uation Agency  (OTEA)  as  a field  operating  agency  under  the  staff  super- 
vision of  the  Assistant  Chief  of  Staff  for  Force  Development  (ACSFOR), 
Department  of  the  Army.  When  the  Army  General  Staff  was  reorganized 
in  May  1974,  OTEA  became  a field  operating  agency  directly  under  the 
Chief  of  Staff  of  the  Army. 

Current  Context. 

In  the  current  context,  OTEA  is  the  Army's  focal  point  for  user 
testing.  OTEA  supports  the  material  acquisition  process  by  exercising 
responsibility  for  all  operational  testing  and  supports  the  force  de- 
velopment process  by  managing  (i.e.,  overseeing)  force  development 
testing  and  experimentation  and  joint  user  testing. 

10 
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OTEA  conducts  and  evaluates  operational  testing  accomplished  on 
all  major  Army  material  systems  (i.e.,  those  which  involve  over  $50 
million  in  research  and  development  funds  or  over  $200  million  in 
procurement  dollars)  and  certain  designated  non-major  systems.  Addi- 
tionally, OTEA  is  the  Army  point  of  contact  with  OSD,  and  is  responsible 
for  the  overall  Army  management  of  OSD-directed  joint  testing  programs. 
OTEA's  management  emphasis  is  to  those  OSD-directed  joint  tests  where 
the  Army  is  designated  the  lead  service. 

Role  of  the  Interface  Betv/een  OTEA  and  The  Army  Joint  Material  Develop- 
ment Community. 

Stated  simply,  the  role  of  the  interface  between  OTEA  and  the  Army 
Material  Development  Community  (AMDC),  as  represented  by  the  US  Army 
Development  and  Readiness  Command  (DARCOM)  (the  successor  to  AMC)  is 
to  facilitate  communications  between  these  organizations.  Further,  be- 
cause of  OTEA's  limited  personnel  resources  and  the  broad  scope  of  its 
mission,  it  is  imperative  that  this  interface  be  as  efficient  as  possible, 
particularly  between  OTEA  and  the  DARCOM  Headquarters  and  Project  Manage- 
ment Office  levels. 

Unfortunately,  as  the  author  has  previously  observed,  these  com- 
munication interfaces  often  appear  to  be  strained  as  a result  of  per- 
ceptual factors  held  by  personnel  at  both  OTEA  and  DARCOM.  The  remainder 
of  this  report  will  thus  be  devoted  to  analyzing  the  reasons  behind 
the  factors  which  give  rise  to  the  interface  problems  between  OTEA, 

DARCOM  Headquarters  and  the  project  management  offices. 
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SECTION  III 

AN  ANALYSIS  OF  THE  REASONS  FOR  INTERFACE  PROBLEMS 


General . 

Opinions  regarding  the  reasons  for  OTEA  - Headquarters,  DARCOM  - 
PMO  interface  problems  were  solicited  from  personnel  who  were  either 
currently  or  recently  assigned  to  one  of  these  three  organizations.  In 
addition,  an  effort  was  made  to  categorize  the  interviews  between  the 
"manager"  and  "worker"  levels  within  each  of  the  respective  entities. 
This  latter  decision  proved  very  beneficial  in  light  of  the  wide  di- 
vergence of  opinion,  regarding  the  respective  subject,  between  these 
two  levels.  The  remainder  of  this  section  of  the  report  is  presented 
in  the  form  of  summary  analyses  of  the  data  obtained  during  the  inter- 
views. 

OTEA. 

Managerial  level  personnel  at  OTEA  generally  cited  five  reasons 
for  OTEA  - Army  Materiel  Development  Community  interface  problems.  They 
were: 

o Historical 

o Situational/environmental 
o OTEA  adherence  to  the  systems  viewpoint 
o Tester  biases 

In  the  eyes  of  the  OTEA  managerial  interviewees  historical  aspects 
appear  to  have  a considerable  impact  on  OTEA-AMDC  relations.  Put 
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bluntly,  there  is,  from  the  OTEA  managerial  view,  a tendency  in  the  AMDC 
to  regard  the  existence  of  OTEA  as  a "slap"  at  the  competence  and  in- 
tegrity of  the  AMDC  to  execute  a good  unbiased  testing  and  evaluation 
program.  Specifically,  they  are  of  the  opinion  that  personnel,  in  the 
Material  Development  Community,  feel  that  OTEA  simply  usurped  the  old 
Engineering  Service  Test,  and  reporting  formerly  performed  by  the  TECOM 
Test  Boards,  and  duplicated  portions  of  the  development  test  cycle.  These 
views,  from  the  OTEA  manager's  standpoint,  are  regarded  as  grossly  in- 
accurate in  light  of  the  agency's  systems  approach  to  testing  (which  is 
discussed  in  the  following  paragraphs). 

Situational /environmental  reasons  for  interface  problems  were  believed 
by  OTEA  managers  to  be  derived  from  the  role  that  OTEA  is  required  by 
charter  to  perform  in  the  material  acquisition  process,  i.e.,  to  make 
an  independent  evaluation  of  the  military  utility,  operational  effective- 
ness and  operational  suitability  of  the  major  and  designated  non-major 
systems  in  development,  and  to  report  directly  to  the  Army  Chief  of 
Staff.  PMO  personnel  have  inferred  that  they  feel  that  OTEA  is  "checking 
on  them"  and  really  cannot  enjoy  a very  comfortable  relationship  with 
anyone  who  is  cast  in  that  role. 

The  third  ingredient  cited  by  the  OTEA  managerial  staff  as  con- 
tributing to  interface  problems  with  the  AMDC  is  OTEA's  adherence  to 
the  "systems"  viewpoint  (as  opposed  to  the  developer's  "subsystem" 
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focus).  From  their  standpoint,  the  material  developers  focus  on  the 
hardware  - technical  aspects  of  testing  with  minimum  regard  for  the 
soldier  operator  and  for  the  non-hardware  subsystems  (e.g.,  training, 
logistics  support).  Operational  testing,  as  executed  by  OTEA,  on  the 
other  hand,  is  believed  to  direct  equal  attention  to  the  relationships 
and  roles  of  all  the  subsystems  and  how  they  impact  on  the  military 
utility,  operational  effectiveness  and  operational  suitability  of  the 
total  system.  This  difference  in  perspective,  or  viewpoint,  is  seen 
as  a key  source  of  misunderstanding  between  OTEA  and  the  AMDC. 

The  fourth  item  cited  by  OTEA  managers  as  a hinderance  to  relations 
with  DARCOM  was  "tester  biases"  within  OTEA  itself.  The  two  specific 
biases  cited  were:  overtesting  and  "finding  something  wrong".  Over- 

testing refers  to  the  tendency  to  conduct  testing  without  regard  to 
time  or  fiscal  constraints  or  consideration  of  the  principle  of  marginal 
returns  as  it  affects  testing  (i.e.,  will  the  value  derived  from  an 
additional  increment  of  testing  justify  the  additional  costs  incurred?). 
The  "finding  something  wrong"  bias  alludes  to  the  tendency  to  feel  that 
an  operational  test  is  unsuccessful  if  it  does  not  uncover  some  major 
system  deficiency.  Managers  interviewed  generally  felt  that  this  is  a 
problem  that  is  associated  with  new  personnel  whose  own  perspectives 
of  operational  testing  are  still  being  formulated.  It  is,  however, 
viewed  as  a very  real  problem  which  the  agency  must  continuously  guard 
against. 
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The  last  item  commonly  cited  by  OTEA  managers  related  to  incon- 
sistencies (i.e.,  the  variation  in  attitudes  toward  OTEA)  shown  by 
people  working  in  the  PMO's.  Specifically  cited  was  the  fact  that  PMO 
personnel  at  the  managerial  levels,  are  generally  cooperative  in  their 
relationships  with  OTEA,  but  lower  level  personnel  usually  are  not.  This 
situation  is  perceived  at  OTEA  as  due  to  either  a breakdown  in  the  in- 
ternal PMO  education  process  or  as  a dichotomous  situation  wherein  the 
PMO  managerial  levels  are  exhibiting  a cooperative  stance  toward  OTEA, 
but  directing  the  working  levels  to  proceed  with  caution  in  their  com- 
munications with  the  operational  testing  community. 

An  analysis  of  the  views  of  interviewees  at  the  non-managerial  levels 
indicated  that  some  overall  differences  in  opinion  did  exist  from  those 
of  the  managerial  group  in  regard  to  their  cognizance  of  the  reasons 
for  OTEA-AMDC  interface  problems.  Key  areas  cited  were:  The  PMO's 

disdain  for  non-advocates  of  their  systems;  failure  to  understand  the 
OT  process;  attempts  by  PM's  to  influence  OTEA;  and  TECOM's  position  in 
the  AMDC  structure. 

The  first  of  the  above  items,  disdain  for  non-advocates  of  their 
systems,  was  frequently  mentioned  by  OTEA  working  level  assignees  as 
an  obvious  element  in  the  attitude  of  PMO  employees  in  their  communi- 
cations with  OTEA.  This  characteristic  was  cited  as  most  prevalent  at 
the  lower  working  levels  within  the  PMOs.  Interviewees  expressed  con- 
cern that  this  attitude  significantly  hinders  communications  between 
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the  OT  and  Material  Development  Community  to  the  detriment  of  the  Army 
as  a whole.  In  a positive  vein,  however,  there  was  a mutual  feeling 
that  this  situation  is  improving  as  new  personnel  replace  those  who 
were  assigned  to  the  PMO  during  the  OTEA  activation  era. 

The  second  item,  failure  to  understand  the  operational  testing 
process,  was  regarded  by  most  respondents  as  probably  the  key  deficiency 
attributing  to  OTEA-AMDC  interface  problems.  Factors  cited  were  the 
lack  of  appreciation  of  the  OT  focus  on  the  systems  viewpoint;  failure 
to  understand  the  generally  lower  confidence  levels  achievable  through 
OT  as  opposed  to  precise  developmental  testing;  seeming  inability  to 
appreciate  the  significance  of  the  timing  factor  in  accomplishing  a good 
operational  test  and  evaluation  (i.e.,  "They  always  want  to  give  OT 
that  time  left  over  after  a very  precise,  meticulous  development  test, 
and  then  expect  a thorough  report  of  testing  immediately  afterwards".). 

Attempts  by  PM's  to  influence  OTEA  were  noted  by  some  respondees 
as  having  a high  potential  for  impeding  OTEA-AMDC  relationships.  Some 
persons  related  overt  incidences  when  these  types  of  attempts  were  allegedly 
made,  but  most  admitted  that  such  efforts  are  usually  subtle.  Personnel 
interviewed  indicated  that  they  expected  the  AMDC  representatives  to 
strongly  respect  their  position  as  independent  testers  and  evaluators. 

The  last  item,  TECOM's  position  in  the  AMDC  structure,  was  re- 
garded by  some  interviewees  as  one  of  the  reasons  for  OTEA  interface 
problems  with  the  AMDC.  Put  simply,  most  interviewees  expressed  a 


fesling  that  TECOM  and  its  independent  evaluation  element.  Army  Material 
Systems  Analysis  Agency  (AMSAA),  cannot  be  regarded  as  truly  independent 
as  long  as  they  remain  a part  of  DARCOM.  Thus,  they  believe  that  OTEA 
cannot  conduct  its  relations  with  TECOM  as  if  it  were  a full  member  of 
the  independent  testing  community. 

Headquarters,  DARCOM 

Interviews  conducted  at  Headquarters,  DARCOM  provided  some  interest- 
ing perspectives  from  a policy  level  viewpoint  on  the  OTEA-AMDC  inter- 
face problem.  Interestingly,  their  comments  somewhat  approximated  the 
responses  of  OTEA  managerial  interviewees.  Their  answers  indicated  the 
following  reasons  as  the  major  contributors  to  the  interface  dilemma: 

o Historical 
o Situational 
o Fiscal 

The  historical  conditions  cited  as  contributing  to  interface  prob- 
lems generally  addressed  the  circumstances  and  conflicts  which  were  rooted 
in  the  OTEA  gestation  period.  Particular  items/events  cited  included: 

The  AMDC's  loss  of  control  over  almost  all  aspects  of  operational  testing; 
the  general  feeling  that  Headquarters,  Department  of  the  Army  succumbed 
to  political  pressure  and  never  gave  the  AMDC  a chance  to  "prove  out" 
the  concept  of  expanded  service  testing;  the  failure  of  OSD,  the  Army 
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Secretariat  and  OTEA  to  appreciate  the  cost  impact  of  incorporating  OT 
into  ongoing  programs;  OTEA's  paranoia  with  regard  to  its  independence 
fc  and  the  general  lack  of  testing  expertise  within  OTEA. 

In  addition  to  the  above  historical  factors,  the  personnel  who 
were  interviewed  cited  current  situational  factors  which  also  detract 
from  OTEA-AMDC  relationships.  Key  among  these  were:  inconsistent  and 

sometimes  conflicting  guidance,  pertaining  to  OT,  from  the  OSD  level; 
the  continuing  assignment  to  OTEA  of  personnel  with  no  prior  experience 
in  Army  testing;  the  general  belief  that  OTEA  is  not  (and  probably  can 
never  be)  truly  independent. 

The  last  of  the  most  frequently  cited  areas  was  fiscal  considerations. 
In  short,  respondees  were  of  the  general  conviction  that  OTEA  still  has 
a "long  way  to  go"  in  learning  to  tailor  its  testing  philosophy  to  the 
relatively  small  amount  of  funds  available  in  the  current  environment. 
Specifically,  in  the  opinion  of  interviewees,  they  must  be  more  willing 
to  conduct  combined  testing,  cross  utilize  the  results  of  developmental 
and  contractor  testing,  and  change  their  negative  attitude  toward  the 
employment  of  simulations. 

Project  Manager  Organizations. 

Interviewees  who  were  either  then  (at  the  time  of  interview)  or 
recently  assigned  to  project  manager  organizations,  generally  cited 
historical  and/or  situational/environmental  and  fiscal  bases  for  their 
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opinions  of  reasons  for  conflict  between  OTEA  and  the  Army  Material 
Development  Community.  Analysis  of  the  results  of  the  interviews 
suggests  that  a wide  gulf  exists  between  the  views  of  the  higher 
level  PM  personnel  (i.e.,  the  project  managers/deputies)  and  their 
subordinates  in  the  PMO's)  in  regard  to  their  views  of  the  extent  and 
nature  of  the  subject  interface  problem. 

Historical  reasons  often  cited  by  higher  level  project  management 
as  contributing  to  the  interface  problem  included  the  PM's  perception 
that  OTEA's  existence  was  (is)  a symbol  of  general  distrust  (of  the  PM) 
by  Congress,  OSD,  and  Headquarters,  Department  of  the  Army.  In  addition, 
however,  most  of  the  project  managers  also  indited  the  actions  of  the  AMDC 
itself  as  a decisive  factor  in  the  establishment  of  OTEA.  Reasons  men- 
tioned included:  the  failure  (of  the  AMDC)  to  press  the  user  for  effective 

operational  test  criteria  against  which  developing  systems  could  have 
been  evaluated,  and  excessive  advocacy  on  the  part  of  material  developers. 

Situational  factors  cited  by  PM's  (and  deputy  PM's)  alluded  to  an 
unrealistic  OTEA  concept  of  the  scope  of  an  effective  operational  test 
("They  want  to  simulate  a small  war".)  and  the  need  to  coordinate  with 
two  separate  testing  organizations  (OTEA  and  TECOM).  Most  of  the  inter- 
viewees expressed  a wish  that  TECOM  and  OTEA  could  somehow  be  combined. 

The  third  area  most  frequently  cited  by  project  and  deputy  project 
managers  was  fiscal,  or  cost,  implications  of  OTEA.  Their  comments 
generally  expressed  a desire  that  OTEA  could  be  funded  to  completely 


19 


I 

' 


- I I II  ■ ' 


supplement  the  real  increases  in  the  PM's  testing  budget  which  are  as- 
sociated with  operational  testing. 

The  responses  elicited  from  interviewees  assigned  at  the  working 
levels  within  the  PMO's  generally  mentioned  the  situational  and  fiscal 
aspects  as  the  key  contributors  to  the  interface  problem.  Comments 
regarding  the  former  included:  a general  recognition  of  the  somewhat 

dichtomous  roles  of  OTEA  and  the  Material  Development  Community;  a desire 
to  combine  OTEA  and  TECOM;  the  belief  that  OTEA  has  too  much  power  in 

I 

the  Army  hierarchy,  and  a conviction  that  there  must  be  a better  overall 
approach  to  OT  than  that  now  pursued  by  the  Army  (several  respondees 
suggested  that  a joint  TRADOC  and  U.  S.  Army  Forces  Command  (FORSCOM) 
approach,  to  operational  testing,  without  OTEA,  would  yield  better  re- 
sults). Other  comments  generally  dealt  with  what  were,  in  the  opinion 
of  interviewees,  believed  to  be  OTEA  deficiencies  which  aggravated  its 
(i.e.,  OTEA's)  relationship  with  the  material  developer.  These  perceived 
shortcomings  were:  a lack  of  appreciation  of  the  complexity  of  the 

PM-contractor  interface,  and  the  potentially  large  financial  and  con- 
tractual impact  of  even  small  design  changes  particularly  in  the  latter 
stages  of  development;  an  overall  low  level  of  technical  expertise  with- 
in OTEA  regarding  the  systems  tested;  insufficient  TRADOC/FORSCOM  support 
during  field  testing  - which  has,  in  effect,  forced  OTEA  to  defend  some 
rather  poorly  executed  tests,  and  too  many  OTEA  personnel  apparently 
authorized  as  agency  spokesmen. 
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Because  the  subject  matter  of  the  foregoing  interviews  was  fo- 
cused at  identifying  the  reasons  for  the  existence  of  problems  in  the 
OTEA-AMDC  interface,  most  of  the  comments  solicited  by  the  interviews 
were  of  a negative  nature.  There  were,  however,  positive  comments 
which  were  repeatedly  encountered  and  bear  mentioning. 

Th?  first  of  these  was  a consensus  --  which  was  expressed  by  most 
Headquarters,  DARCOM  and  PM  interviewees  in  higher  levels  of  management, 
regarding  the  need  for  an  independent  operational  test  organization  such 
as  OTEA.  Most  of  the  respondees  approved  of  the  idea  of  an  independent 
operational  test  agency  citing  it  as  an  asset  in  obtaining  both  political 
and  economic  support  for  their  respective  systems  (even  though  in  pratice 
they  would  prefer  a merger  of  OTEA  and  TECOM  into  a single  independent 
testing  agency). 

A second  development  frequently  cited  was  the  positive  turn  in  OTEA 
PMO  working  relationships  which  has  resulted  from  test  integration  work- 
ing groups.  (TIWGs) . Interviewees  at  OTEA,  HQ,  DARCOM,  and  at  the  PMO's 
seemed  to  welcome  this  change  and  perceived  it  as  a sign  of  maturation 
between  the  OT  and  Material  Development  Communities. 
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SYNTHESIS  AND  EVALUATION 

There  appears  to  be  a general  recognition  within  OTEA,  Headquarters, 
DARCOM,  and  the  project  management  offices  that  considerable  inter- 
face problems  still  exist  between  OTEA  and  the  AMDC.  Both  parties  ac- 
knowledge that  these  problems  have  historical  and  situational  bases 
which  are  not  easily  rectifiable. 

At  the  managerial  levels,  where  personnel  are  more  familiar  with 
the  roles  their  organizations  fulfill  in  the  Army  material  acquisition 
cycle,  there  is  considerable  optimism  regarding  present  trends  in  OTEA- 
AMDC  relations  and  the  possibility  of  further  reducing  current  inter- 
face problems.  In  contrast,  the  perceptions  of  personnel  at  the  working 
levels  are  still  characterized  by  suspicion  and  parochialism. 
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CONCLUSIONS  AND  RECOMMENDATIONS 

Conclusions . 

Current  interface  problems  between  OTEA  and  the  Army  Material  De- 
velopment Community  are  due  primarily  to  negative  perceptions  of  one 
another  held  by  personnel  at  the  working  levels  of  these  two  entities. 

These  negative  ideas  are  rooted  in  the  conflict  which  surrounded  the 
genesis  of  OTEA  and  are  perpetuated  by  the  differing  roles  which  OTEA 
and  the  AMDC  organizations  are  assigned  in  the  material  acquisition 
process. 

Recommendations. 

There  appears  to  be  a definite  need  at  OTEA  and  DARCOM,  particu- 
larly at  the  project  manager  office  level  of  the  latter,  to  insure  that 
working-level  personnel  are  better  educated  in  regard  to  the  overall 
Army  material  acquisition  process  and  the  roles  of  their  respective  or- 
ganizations. Steps  for  implementing  this  recommendation  include: 

- Establishing  in-house  training  classes,  where  none  are  currently 
held,  with  the  goal  of  broadening  the  general  knowledge  of  working-level 
personnel  in  regard  to  the  material  acquisition  process  and  their  roles 

in  it. 

- Expanding  the  scope  of  existing  in-house  training  classes 

to  provide  a better  understanding  of  the  role  of  one's  own  organization 
in  the  comprehensive  Army  material  acquisition  process. 
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- Insuring  that  the  viewpoints  of  the  "other  side  of  the  house" 
i.e.,  operational  testing  or  material  development,  are  amply  presented 
at  these  training  sessions,  (e.g.,  inviting  a speaker  from  OTEA  to 
make  a presentation  at  PMO  training  sessions  and  vice  versa.) 

- Monitoring  the  assignment  of  new  personnel  to  insure  that 
both  OTEA  and  the  PMOs  are  receiving  a fair  share  of  personnel  who  are 
participating  in  the  Army  Project  Management  Development  Program  career 
field. 


- Insuring  that  assigned  personnel,  both  civilian  and  military, 
are  provided  an  opportunity  to  attend  one  of  the  Program  Management 
Courses  offered  by  the  Defense  Systems  Management  School,  if  they  so 
desire. 


24 


APPENDIX  A 


DAPCOM  - PROJECT  MANAGER  QUESTIONNAIRE 

1.  Job  - position/how  long  held?  Other  jobs  in  material  organizations? 

2.  What  is  your  opinion  regarding  the  need  for  OTEA? 

3.  Was  the  DT  expanded  service  test  procedure,  which  preceeded  the 
current  OTEA  directed  approach  to  OT,  better? 

4.  Has  your  project/office  interfaced  with  OTEA?  If  so,  to  what  ex- 
tent and  how  frequently? 

5.  What  element  in  your  PMO  normally  deals  with  OTEA? 

6.  What  level  in  OTEA  do  they  normally  interface  with? 

7.  Have  you,  personally,  had  any  business  dealings  with  OTEA?  If  so, 
what  element  do  you  normally  communicate  with  and  how  frequently? 

8.  What  have  been  your  key  periods  of  interface  with  OTEA? 

9.  Have  you  found  that  OTEA's  organization  structure  hindered  your  re- 
lations with  it? 

10.  Is  OTEA  large  enough  to  handle  its  responsibilities? 

11.  How  would  you  characterize  your  relationship  with  OTEA? 

12.  Have  any  general  interface  type  problems  arisen? 

13.  Have  OTEA's  demands  upon  your  PM  organization  been  reasonable? 

14.  Has  OTEA  been  responsive  to  your  requests? 

15.  In  your  opinion,  does  OTEA  "pay  its  own  way"  or  have  you  had  to 
divert  resources  to  support  their  testing? 

16.  Has  OTEA  had  any  impact  on  the  way  your  project  is  managed?  e.g., 
cost,  scheduling,  additional  manpower? 
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17.  Has  OTEA  caused  any  problems  between  you  and  your  major  command 
or  higher  headquarters? 

18.  Has  OTEA  affected  or  impacted  your  relationship  with  contractors? 

19.  How  would  you  characterize  OTEA/PM/HQ,  DARCOM  communications?  Is 
there  a communication  gap? 

(If  answer  to  19  is  positive) 

20.  What  do  you  suggest  to  overcome? 

- LNO's? 

- More  frequent  meetings? 

- More  combined  testing? 

- Other? 

21.  Do  you  feel  that  OTEA  has  too  much  power  in  the  Army  hierarchy? 
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OTEA  QUESTIONNAIRE 

1.  Job/position/how  long/previous  assignments  in  material  acquisition? 

2.  Which  PMO's  have  you  interfaced  with?  If  so,  to  what  extent? 

3.  How  would  you  characterize  your  relationships  with  each  of  these  PMO's? 

4.  What  element  in  the  PMO's  did  you  normally  interact  with? 

5.  What  have  been  your  key  periods  of  interface? 

6.  Have  you  found  that  the  structure  of  the  various  PMO’s  hindered  your  ] 

relationships  with  them? 

7.  Have  any  significant  "interface  type"  problems  arisen? 

8.  Have  the  PMO's  demands  on  OTEA  been  reasonable? 

9.  Have  the  PMO's  been  responsive  to  OTEA's  requests? 

10.  How  would  you  characterize  OTEA/PM  communications?  Is  there  a com- 
munications gap?  j 

11.  Do  OTEA/PM  communications  need  to  be  facilitated?  If  so,  how? 

- LNO's? 

- More  frequent  meetings? 

- More  combined  testing? 

- Other? 

12.  Have  the  PMO's  aired  any  complaints  to  you  regarding  what  they  visualize 
as  interface  problems  with  OTEA? 

13.  To  what  extent,  if  any,  have  you  been  required  to  work  directly  with 
Headquarters,  DARCOM?  What  element(s)  do  you  normally  work  with? 

14.  How  would  you  characterize  your  relationship(s)  with  HQ,  DARCOM? 
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